Exploring Medalla data

Initial exploration.

Barnabé Monnot https://twitter.com/barnabemonnot (Robust Incentives Group, Ethereum Foundation)https://github.com/ethereum/rig
2020-10-12

Table of Contents


In this notebook we explore data from the Medalla testnet. We are looking at the 20000 first slots.

Data sources

Lighthouse block export

We use a fork of Lakshman Sankar’s Lighthouse block exporter to export attestations and blocks from the finalised chain until slot 20000.

We present the main datasets below:

logical_ats

Each row in this dataset corresponds to an aggregate attestation included in a block. We cast its attesting_indices (the bits recording which validators are aggregated in this aggregate attestation) to a wide format, with one column per validator. This is shown below (columns past column “3” are truncated).

slot att_slot committee_index beacon_block_root source_block_root target_block_root 1 2 3
1 0 0 3ca1a0660c 0000000000 3ca1a0660c TRUE FALSE FALSE
1 0 0 3ca1a0660c 0000000000 3ca1a0660c FALSE FALSE FALSE
1 0 0 3ca1a0660c 0000000000 3ca1a0660c FALSE FALSE FALSE
1 0 0 3ca1a0660c 0000000000 3ca1a0660c FALSE FALSE FALSE
3 0 0 3ca1a0660c 0000000000 3ca1a0660c TRUE FALSE FALSE
12 0 0 3ca1a0660c 0000000000 3ca1a0660c TRUE FALSE FALSE

exploded_ats

We cast the dataset above into a long format, such that each row corresponds to an individual attestation included in a block. Note that when this individual attestation is included multiple times over multiple aggregates, it appears multiple times in the dataset.

slot att_slot committee_index beacon_block_root source_block_root target_block_root index_in_committee
1 0 0 3ca1a0660c 0000000000 3ca1a0660c 0
3 0 0 3ca1a0660c 0000000000 3ca1a0660c 0
12 0 0 3ca1a0660c 0000000000 3ca1a0660c 0
14 0 0 3ca1a0660c 0000000000 3ca1a0660c 0
1 0 0 3ca1a0660c 0000000000 3ca1a0660c 4
3 0 0 3ca1a0660c 0000000000 3ca1a0660c 4

all_dps

Validators are allowed to deposit more ETH into their eth2 accounts. Later on, we compute the reward obtained by a validator by comparing its initial balance with its current balance. We must then deduct deposits that were made since genesis.

slot pubkey amount validator_index
7999 0x8118e8008f3d0d0e9e55104bc8fe02f38beec59b470f14ab6a52300d3c403e55e54efe6b00efbc9b8e93d4e31680d37c 32000000000 24682
7999 0xb757c0eedd579da096150d865142b8a9504f79ac169e3bf7a05b7af5ef7b5a98f5c1ef9cff4df232b5b28e52a25b548d 32000000000 24683
7999 0x972a30ea9b0c8689e7d1ba24332cc2e927c1dd38105a0e19b5d597c70a5c48be499afd0f0c254fa1c55f53d2f77fa72b 32000000000 24684
7999 0xaa65a69d4fe1fad7696f26bc086e755e1d8e1e63e91ee690a7a201d2e8f06ca7feffd93167607293e4b70d8189c44fff 32000000000 24685
7999 0xab2fa95d5c929622ec620cbc831a300868281e37f5cb8d6b22a903c7b8d332acfc13d7961058e53bb03a1f15462de847 32000000000 24686
7999 0x895838e4800a571b9b74cec5a82c59f47dff2a5666fe8f5aee5500142b5734b0942afe6606cfca35944208b467828fce 32000000000 24687

Weald dump

Jim McDonald, from Attestant, kindly provided a treasure trove of data on the #medalla-data-challenge channel of the EthStaker Discord server. The two previous datasets could have legitimately been mined from Jim’s data, but we like to get our hands dirty.

all_cms

Not too dirty though: obtaining the past record of committees (which validators are supposed to attend when) is much more computationally intensive, since it requires access to past states. This dataset is a long-format record of which validators appear in which committee and when.

att_slot committee_index index_in_committee validator_index epoch
0 0 0 2070 0
0 0 1 15688 0
0 0 2 14974 0
0 0 3 16691 0
0 0 4 10370 0
0 0 5 11439 0

val_balances

The final dataset gives us validator state balances at the end of each epoch. Note that the state balance, the true ETH amount a validator owns, is different from effective balances, which measure the principal on which validators receive an interest.

epoch validator_index balance
0 4748 32000000000
0 5881 32000000000
0 12766 32000000000
0 13679 32000000000
0 16759 32000000000
0 3594 32000000000

Validator performance

Validators are rewarded for their performance, and penalised for failing to complete their tasks. We start with a crude measure of performance: the number of included attestations. It is a crude measure since (a) we do not discount the timeliness of the validator, measured by the inclusion delay and (b) we do not check that the attestation’s attributes are correct (with the exception of the source attribute, since an incorrect source cannot possibly be included on-chain).

Uptime-rewards curve I: Included attestations

We compare the percentage of included attestations with the (possibly negative) reward obtained by the validator.

Uptime-rewards curve II: Inclusion delay

We turn our attention to the inclusion delay. Validators are rewarded for attesting timely, with higher rewards the earlier they are included in a block. We explode aggregates contained in the blocks to trace the earliest included attestation of each validator in an epoch.

Note that the y axis is given on a logarithmic scale. A high number of attestations have a low inclusion delay, which is good! Since attestations cannot be included more than 32 slots from their attesting slot, the distribution above is naturally capped at 32.

How is the inclusion delay correlated with the rewards? We look at validators who have the highest number of included attestations to find out.

We find a somewhat negative relationship between average inclusion delay and the rewards obtained by validators, with most points concentrated on a single horizontal line.

Aggregate attestations

eth2 is built to scale to tens of thousands of validators. This introduces overhead from message passing (and inclusion) when these validators are asked to vote on the canonical chain. To alleviate the beacon chain, votes (a.k.a. attestations) can be aggregated.

In particular, an attestation contains four attributes:

Since we expect validators to broadly agree in times of low latency, we also expect a lot of redundant attestations. We can aggregate such a set of attestations \(I\) into a single, aggregate, attestation.

For each slot \(s\), a committee of validator \(C(s)\) is determined who is expected to attest for \(s\). Assume that two aggregate attestations were formed from validators attesting for \(s\), one aggregate of validators in set \(I \subseteq C(s)\) and the other with validators in set \(J \subseteq C(s)\). We have two cases:

In the following, we look at redundant, clashing and individual attestations.

How many individual attestations are contained in aggregates?

A fairly high number of aggregate attestations included in a block are actually individual attestations. Nonetheless, a significant number of aggregates tally up between 50 and 100 attestations.

We can plot the same, weighing by the size of the validator set in the aggregate, to count how many individual attestations each size of aggregates included.

Overall, we can plot the Lorenz curve of aggregate attestations. This allows us to find out the share of attestations held by the 20% largest aggregates.

The answer is 39%.

How much savings did aggregates provide?

We compare how many individual attestations exist to how many aggregates were included.

How many distinct individual attestations are there?

Count
10284874

And how many aggregates were included in blocks?

Count
928836

How much savings is that? The ratio of the first to the second.

Savings
11.07286324

We have 11.07 times more individual attestations than aggregates, meaning that if we were not aggregating, we would have 11.07 as much data on-chain.

In how many aggregate attestations is a single attestation included?

We look at all individual attestations in our dataset, i.e., individual, unaggregated votes, and measure how many times they were included in an aggregate.

How many redundant aggregate attestations are there?

We call redundant identical aggregate attestations (same four attributes and same set of validator indices) which are included in more than one block. It can happen when a block producer does not see that an aggregate was previously included (e.g., because of latency), or simply when the block producer doesn’t pay attention and greedily adds as many aggregates as they know about.

The mode is 1, which is also the optimal case. A redundant aggregate does not have much purpose apart from bloating the chain.

How many times did a block include the exact same aggregate attestation more than once?

We could call these strongly redundant, as this is pure waste.

n count
1 928722
2 57

We see that 57 times, identical aggregates were included twice in a block.

How many times were clashing attestations included in blocks?

We define clashing attestations as two aggregate attestations included in the same block, with identical attributes (same attesting slot, beacon chain head, source block and target block). We can further define the following two notions, assuming the two aggregate attestations include attestations of validator sets \(I\) and \(J\) respectively:

We obtain how many times an attestation weakly clashes with itself, i.e., is included multiple times in a single block with different validator sets. For instance, if an aggregate attestation is included in the same block three times with a different set of validator indices each time, we record that this aggregate is weakly clashing three times with itself. We give below the histogram of this measure.

From the plot above, we observe that some aggregates were included over 40 times in the same block, all with different sets of validator indices. Still, most aggregates were included once or a few times.

Finding weakly clashing attestations that are not strongly clashing (i.e., which could have been aggregated further) is left for future work as it is more computationally intensive. In particular, for a set of aggregates identical up to their validator indices, one must find which have an empty overlap.

Note that optimally aggregating a set of aggregates is NP-complete! Here is a reduction of the optimal aggregation problem to the graph colouring. Set aggregate attestations as vertices in a graph, with an edge drawn between two vertices if the validator sets of the two aggregates have a non-empty overlap. In the graph colouring, we look for the minimum number of colours necessary to assign a colour to each vertex such that two connected vertices do not have the same colour. All vertices who share the same colour have an empty overlap, and thus can be combined into an aggregate. The minimum number of colours necessary to colour the graph tells us how few aggregates were necessary to combine a given set of aggregates further.